PART:1 (EDA & DATA PREPROCESSING ON GOOGLE APP STORE RATING DATASET¶

QUESTION 1:¶

Import required libraries and read the dataset.

SOLUTION:¶

In [2]:
import pandas as pd # required libraries imported
import numpy as np
from sklearn.preprocessing import OneHotEncoder
from sklearn.preprocessing import LabelEncoder
import plotly.express as px
lc=pd.read_csv("Apps_data+(1) (1).csv")
lc
Out[2]:
App Category Rating Reviews Size Installs Type Price Content Rating Genres Last Updated Current Ver Android Ver
0 Photo Editor & Candy Camera & Grid & ScrapBook ART_AND_DESIGN 4.1 159 19M 10,000+ Free 0 Everyone Art & Design January 7, 2018 1.0.0 4.0.3 and up
1 Coloring book moana ART_AND_DESIGN 3.9 967 14M 500,000+ Free 0 Everyone Art & Design;Pretend Play January 15, 2018 2.0.0 4.0.3 and up
2 U Launcher Lite – FREE Live Cool Themes, Hide ... ART_AND_DESIGN 4.7 87510 8.7M 5,000,000+ Free 0 Everyone Art & Design August 1, 2018 1.2.4 4.0.3 and up
3 Sketch - Draw & Paint ART_AND_DESIGN 4.5 215644 25M 50,000,000+ Free 0 Teen Art & Design June 8, 2018 Varies with device 4.2 and up
4 Pixel Draw - Number Art Coloring Book ART_AND_DESIGN 4.3 967 2.8M 100,000+ Free 0 Everyone Art & Design;Creativity June 20, 2018 1.1 4.4 and up
... ... ... ... ... ... ... ... ... ... ... ... ... ...
10836 Sya9a Maroc - FR FAMILY 4.5 38 53M 5,000+ Free 0 Everyone Education July 25, 2017 1.48 4.1 and up
10837 Fr. Mike Schmitz Audio Teachings FAMILY 5.0 4 3.6M 100+ Free 0 Everyone Education July 6, 2018 1.0 4.1 and up
10838 Parkinson Exercices FR MEDICAL NaN 3 9.5M 1,000+ Free 0 Everyone Medical January 20, 2017 1.0 2.2 and up
10839 The SCP Foundation DB fr nn5n BOOKS_AND_REFERENCE 4.5 114 Varies with device 1,000+ Free 0 Mature 17+ Books & Reference January 19, 2015 Varies with device Varies with device
10840 iHoroscope - 2018 Daily Horoscope & Astrology LIFESTYLE 4.5 398307 19M 10,000,000+ Free 0 Everyone Lifestyle July 25, 2018 Varies with device Varies with device

10841 rows × 13 columns

QUESTION 2:¶

Check the first few samples, shape, info of the data and try to familiarize yourself with different features.

SOLUTION:¶

In [3]:
lc.sample(5)  #checking some random samples
Out[3]:
App Category Rating Reviews Size Installs Type Price Content Rating Genres Last Updated Current Ver Android Ver
6295 Learn English Vocabulary - 6,000 Words FAMILY 4.5 205914 54M 10,000,000+ Free 0 Everyone Education August 3, 2018 5.54 4.1 and up
7307 CG FM FAMILY 5.0 2 6.6M 500+ Free 0 Everyone Entertainment January 31, 2018 1.0.9 4.1 and up
726 Game for KIDS: KIDS match'em EDUCATION 4.4 7005 6.9M 1,000,000+ Free 0 Everyone Education October 23, 2016 1.17 2.3 and up
7020 Butterfly Pixel Art - coloring by number FAMILY 4.6 769 13M 100,000+ Free 0 Everyone Entertainment April 4, 2018 1.06 4.1 and up
4148 Granny GAME 4.5 1128805 59M 50,000,000+ Free 0 Teen Arcade May 29, 2018 1.4.0.1 4.0 and up
In [4]:
lc.shape  #verifying shape by rows and columns
Out[4]:
(10841, 13)
In [5]:
lc.info
Out[5]:
<bound method DataFrame.info of                                                      App             Category  \
0         Photo Editor & Candy Camera & Grid & ScrapBook       ART_AND_DESIGN   
1                                    Coloring book moana       ART_AND_DESIGN   
2      U Launcher Lite – FREE Live Cool Themes, Hide ...       ART_AND_DESIGN   
3                                  Sketch - Draw & Paint       ART_AND_DESIGN   
4                  Pixel Draw - Number Art Coloring Book       ART_AND_DESIGN   
...                                                  ...                  ...   
10836                                   Sya9a Maroc - FR               FAMILY   
10837                   Fr. Mike Schmitz Audio Teachings               FAMILY   
10838                             Parkinson Exercices FR              MEDICAL   
10839                      The SCP Foundation DB fr nn5n  BOOKS_AND_REFERENCE   
10840      iHoroscope - 2018 Daily Horoscope & Astrology            LIFESTYLE   

       Rating Reviews                Size     Installs  Type Price  \
0         4.1     159                 19M      10,000+  Free     0   
1         3.9     967                 14M     500,000+  Free     0   
2         4.7   87510                8.7M   5,000,000+  Free     0   
3         4.5  215644                 25M  50,000,000+  Free     0   
4         4.3     967                2.8M     100,000+  Free     0   
...       ...     ...                 ...          ...   ...   ...   
10836     4.5      38                 53M       5,000+  Free     0   
10837     5.0       4                3.6M         100+  Free     0   
10838     NaN       3                9.5M       1,000+  Free     0   
10839     4.5     114  Varies with device       1,000+  Free     0   
10840     4.5  398307                 19M  10,000,000+  Free     0   

      Content Rating                     Genres      Last Updated  \
0           Everyone               Art & Design   January 7, 2018   
1           Everyone  Art & Design;Pretend Play  January 15, 2018   
2           Everyone               Art & Design    August 1, 2018   
3               Teen               Art & Design      June 8, 2018   
4           Everyone    Art & Design;Creativity     June 20, 2018   
...              ...                        ...               ...   
10836       Everyone                  Education     July 25, 2017   
10837       Everyone                  Education      July 6, 2018   
10838       Everyone                    Medical  January 20, 2017   
10839     Mature 17+          Books & Reference  January 19, 2015   
10840       Everyone                  Lifestyle     July 25, 2018   

              Current Ver         Android Ver  
0                   1.0.0        4.0.3 and up  
1                   2.0.0        4.0.3 and up  
2                   1.2.4        4.0.3 and up  
3      Varies with device          4.2 and up  
4                     1.1          4.4 and up  
...                   ...                 ...  
10836                1.48          4.1 and up  
10837                 1.0          4.1 and up  
10838                 1.0          2.2 and up  
10839  Varies with device  Varies with device  
10840  Varies with device  Varies with device  

[10841 rows x 13 columns]>

QUESTION 3:¶

Check summary statistics of the dataset. List out the columns that need to be worked upon for model building.

SOLUTION:¶

In [6]:
lc.describe(include='all')  #checking summary stat
Out[6]:
App Category Rating Reviews Size Installs Type Price Content Rating Genres Last Updated Current Ver Android Ver
count 10841 10841 9367.000000 10841 10841 10841 10840 10841 10840 10841 10841 10833 10838
unique 9660 34 NaN 6002 462 22 3 93 6 120 1378 2832 33
top ROBLOX FAMILY NaN 0 Varies with device 1,000,000+ Free 0 Everyone Tools August 3, 2018 Varies with device 4.1 and up
freq 9 1972 NaN 596 1695 1579 10039 10040 8714 842 326 1459 2451
mean NaN NaN 4.193338 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
std NaN NaN 0.537431 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
min NaN NaN 1.000000 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
25% NaN NaN 4.000000 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
50% NaN NaN 4.300000 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
75% NaN NaN 4.500000 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
max NaN NaN 19.000000 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
In [8]:
lc.select_dtypes(include='object').columns.tolist() #catageroies columns, TO check before building the model
Out[8]:
['App',
 'Category',
 'Reviews',
 'Size',
 'Installs',
 'Type',
 'Price',
 'Content Rating',
 'Genres',
 'Last Updated',
 'Current Ver',
 'Android Ver']
In [11]:
lc.columns[lc.isna().any()].tolist()  #before model building nan columns to be worked
Out[11]:
['Rating', 'Type', 'Content Rating', 'Current Ver', 'Android Ver']

QUESTION 4:¶

Check if there are any duplicate records in the dataset? if any drop them.

SOLUTION:¶

In [12]:
lc1=lc.copy()  #making copy
In [13]:
lc[lc.duplicated()]
Out[13]:
App Category Rating Reviews Size Installs Type Price Content Rating Genres Last Updated Current Ver Android Ver
229 Quick PDF Scanner + OCR FREE BUSINESS 4.2 80805 Varies with device 5,000,000+ Free 0 Everyone Business February 26, 2018 Varies with device 4.0.3 and up
236 Box BUSINESS 4.2 159872 Varies with device 10,000,000+ Free 0 Everyone Business July 31, 2018 Varies with device Varies with device
239 Google My Business BUSINESS 4.4 70991 Varies with device 5,000,000+ Free 0 Everyone Business July 24, 2018 2.19.0.204537701 4.4 and up
256 ZOOM Cloud Meetings BUSINESS 4.4 31614 37M 10,000,000+ Free 0 Everyone Business July 20, 2018 4.1.28165.0716 4.0 and up
261 join.me - Simple Meetings BUSINESS 4.0 6989 Varies with device 1,000,000+ Free 0 Everyone Business July 16, 2018 4.3.0.508 4.4 and up
... ... ... ... ... ... ... ... ... ... ... ... ... ...
8643 Wunderlist: To-Do List & Tasks PRODUCTIVITY 4.6 404610 Varies with device 10,000,000+ Free 0 Everyone Productivity April 6, 2018 Varies with device Varies with device
8654 TickTick: To Do List with Reminder, Day Planner PRODUCTIVITY 4.6 25370 Varies with device 1,000,000+ Free 0 Everyone Productivity August 6, 2018 Varies with device Varies with device
8658 ColorNote Notepad Notes PRODUCTIVITY 4.6 2401017 Varies with device 100,000,000+ Free 0 Everyone Productivity June 27, 2018 Varies with device Varies with device
10049 Airway Ex - Intubate. Anesthetize. Train. MEDICAL 4.3 123 86M 10,000+ Free 0 Everyone Medical June 1, 2018 0.6.88 5.0 and up
10768 AAFP MEDICAL 3.8 63 24M 10,000+ Free 0 Everyone Medical June 22, 2018 2.3.1 5.0 and up

483 rows × 13 columns

In [14]:
lc.drop_duplicates(inplace=True)
In [15]:
lc.shape # The number of rows reduced
Out[15]:
(10358, 13)

QUESTION 5:¶

Check the unique categories of the column 'Category', Is there any invalid category? If yes, drop them

SOLUTION:¶

In [17]:
lc['Category'].unique()
Out[17]:
array(['ART_AND_DESIGN', 'AUTO_AND_VEHICLES', 'BEAUTY',
       'BOOKS_AND_REFERENCE', 'BUSINESS', 'COMICS', 'COMMUNICATION',
       'DATING', 'EDUCATION', 'ENTERTAINMENT', 'EVENTS', 'FINANCE',
       'FOOD_AND_DRINK', 'HEALTH_AND_FITNESS', 'HOUSE_AND_HOME',
       'LIBRARIES_AND_DEMO', 'LIFESTYLE', 'GAME', 'FAMILY', 'MEDICAL',
       'SOCIAL', 'SHOPPING', 'PHOTOGRAPHY', 'SPORTS', 'TRAVEL_AND_LOCAL',
       'TOOLS', 'PERSONALIZATION', 'PRODUCTIVITY', 'PARENTING', 'WEATHER',
       'VIDEO_PLAYERS', 'NEWS_AND_MAGAZINES', 'MAPS_AND_NAVIGATION',
       '1.9'], dtype=object)
In [20]:
Ineffectual = lc[lc["Category"] == "1.9"]
In [21]:
Ineffectual
Out[21]:
App Category Rating Reviews Size Installs Type Price Content Rating Genres Last Updated Current Ver Android Ver
10472 Life Made WI-Fi Touchscreen Photo Frame 1.9 19.0 3.0M 1,000+ Free 0 Everyone NaN February 11, 2018 1.0.19 4.0 and up NaN
In [22]:
lc.drop(10472,inplace=True)  #putting Ineffectual value row corresponding to catagery column.
In [23]:
lc.shape
Out[23]:
(10357, 13)

QUESTION 6:¶

Check if there are missing values present in the column Rating, If any? drop them and and create a new column as 'Rating_category' by converting ratings to high and low categories(>3.5 is high rest low)

SOLUTION:¶

In [24]:
lc[lc['Rating'].isnull()==True].index  #missing rating of index row
Out[24]:
Int64Index([   23,   113,   123,   126,   129,   130,   134,   163,   180,
              185,
            ...
            10816, 10818, 10821, 10822, 10823, 10824, 10825, 10831, 10835,
            10838],
           dtype='int64', length=1465)
In [25]:
lc.drop(lc[lc['Rating'].isnull()==True].index,inplace=True)
In [26]:
lc.shape  #now null rating is shown
Out[26]:
(8892, 13)
In [27]:
#Creating a function to categorize the ratings.

def Rating_category(value):
    if value <= 3.5:
        return "Low"
    elif value > 3.5:
        return "High"
In [28]:
lc["Rating_category"] = lc['Rating'].apply(Rating_category)
In [29]:
lc
Out[29]:
App Category Rating Reviews Size Installs Type Price Content Rating Genres Last Updated Current Ver Android Ver Rating_category
0 Photo Editor & Candy Camera & Grid & ScrapBook ART_AND_DESIGN 4.1 159 19M 10,000+ Free 0 Everyone Art & Design January 7, 2018 1.0.0 4.0.3 and up High
1 Coloring book moana ART_AND_DESIGN 3.9 967 14M 500,000+ Free 0 Everyone Art & Design;Pretend Play January 15, 2018 2.0.0 4.0.3 and up High
2 U Launcher Lite – FREE Live Cool Themes, Hide ... ART_AND_DESIGN 4.7 87510 8.7M 5,000,000+ Free 0 Everyone Art & Design August 1, 2018 1.2.4 4.0.3 and up High
3 Sketch - Draw & Paint ART_AND_DESIGN 4.5 215644 25M 50,000,000+ Free 0 Teen Art & Design June 8, 2018 Varies with device 4.2 and up High
4 Pixel Draw - Number Art Coloring Book ART_AND_DESIGN 4.3 967 2.8M 100,000+ Free 0 Everyone Art & Design;Creativity June 20, 2018 1.1 4.4 and up High
... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
10834 FR Calculator FAMILY 4.0 7 2.6M 500+ Free 0 Everyone Education June 18, 2017 1.0.0 4.1 and up High
10836 Sya9a Maroc - FR FAMILY 4.5 38 53M 5,000+ Free 0 Everyone Education July 25, 2017 1.48 4.1 and up High
10837 Fr. Mike Schmitz Audio Teachings FAMILY 5.0 4 3.6M 100+ Free 0 Everyone Education July 6, 2018 1.0 4.1 and up High
10839 The SCP Foundation DB fr nn5n BOOKS_AND_REFERENCE 4.5 114 Varies with device 1,000+ Free 0 Mature 17+ Books & Reference January 19, 2015 Varies with device Varies with device High
10840 iHoroscope - 2018 Daily Horoscope & Astrology LIFESTYLE 4.5 398307 19M 10,000,000+ Free 0 Everyone Lifestyle July 25, 2018 Varies with device Varies with device High

8892 rows × 14 columns

QUESTION 7:¶

Check the distribution of the newly created column 'Rating_category' and comment on the distribution.

SOLUTION:¶

In [30]:
distributing = lc["Rating_category"].value_counts()
In [31]:
distributing
Out[31]:
High    8012
Low      880
Name: Rating_category, dtype: int64
In [32]:
#Plotting histogram to see the distrubution
lc["Rating_category"].hist()
Out[32]:
<Axes: >

COMMENT OF THE GRAPH:¶

The majority of apps have ratings exceeding 3.5, while only a small portion of apps have ratings below 3.5.

QUESTION 8:¶

Convert the column "Reviews'' to numeric data type and check the presence of outliers in the column and handle the outliers using a transformation approach.(Hint: Use log transformation)

In [33]:
lc['Reviews']=lc['Reviews'].astype('int')   #data type conversion
In [34]:
lc['Reviews'].dtypes
Out[34]:
dtype('int32')
In [36]:
import plotly.express as px
fig = px.box(lc['Reviews'])
fig.show()                     #dots are says outliers present
In [38]:
#We applying log trasformation

log10 = np.log10(lc["Reviews"])
In [39]:
log10.describe()
Out[39]:
count    8892.000000
mean        3.576368
std         1.685149
min         0.000000
25%         2.214844
50%         3.673435
75%         4.852887
max         7.892975
Name: Reviews, dtype: float64
In [40]:
fig = px.box(log10)
fig.show()   #make box plot, without outliers clear boxplot
In [41]:
lc["Reviews"] = log10

QUESTION 9:¶

The column 'Size' contains alphanumeric values, treat the non numeric data and convert the column into suitable data type. (hint: Replace M with 1 million and K with 1 thousand, and drop the entries where size='Varies with device')

SOLUTION:¶

In [42]:
lc['Size']
Out[42]:
0                       19M
1                       14M
2                      8.7M
3                       25M
4                      2.8M
                ...        
10834                  2.6M
10836                   53M
10837                  3.6M
10839    Varies with device
10840                   19M
Name: Size, Length: 8892, dtype: object
In [43]:
lc["Size"] = lc["Size"].apply(lambda x : x.replace(",",""))  #replacing the comma
In [44]:
# changing the  alphabetical to numerical
lc["Size"] = lc["Size"].str.replace("M","000000")
In [45]:
lc["Size"] = lc["Size"].str.replace("k","000")
In [46]:
lc[lc['Size']=='Varies with device'].index
Out[46]:
Int64Index([   37,    42,    52,    67,    68,    73,    85,    88,    89,
               92,
            ...
            10647, 10679, 10681, 10707, 10712, 10713, 10725, 10765, 10826,
            10839],
           dtype='int64', length=1468)
In [47]:
lc.drop(lc[lc['Size']=='Varies with device'].index,inplace=True)
In [48]:
lc.shape
Out[48]:
(7424, 14)
In [49]:
#Converting the "SIZE" column from an Object data type to the most appropriate data type
lc["Size"].convert_dtypes()
Out[49]:
0         19000000
1         14000000
2        8.7000000
3         25000000
4        2.8000000
           ...    
10833       619000
10834    2.6000000
10836     53000000
10837    3.6000000
10840     19000000
Name: Size, Length: 7424, dtype: string

QUESTION 10:¶

Check the column 'Installs', treat the unwanted characters and convert the column into a suitable data type.

In [50]:
lc['Installs'] # the unlike symbols are '+',",".  we need to remove them
Out[50]:
0            10,000+
1           500,000+
2         5,000,000+
3        50,000,000+
4           100,000+
            ...     
10833         1,000+
10834           500+
10836         5,000+
10837           100+
10840    10,000,000+
Name: Installs, Length: 7424, dtype: object
In [51]:
#after removing the unwant symbols , the dataset
lc["Installs"] = lc["Installs"].str.replace("+","")

C:\Users\SUNILKUMARAV\AppData\Local\Temp\ipykernel_24020\654164894.py:2: FutureWarning:

The default value of regex will change from True to False in a future version. In addition, single character regular expressions will *not* be treated as literal strings when regex=True.
In [52]:
lc["Installs"]
Out[52]:
0            10,000
1           500,000
2         5,000,000
3        50,000,000
4           100,000
            ...    
10833         1,000
10834           500
10836         5,000
10837           100
10840    10,000,000
Name: Installs, Length: 7424, dtype: object
In [53]:
#Converting the "Installs" column from an Object data type to the most suitable data type.
lc["Installs"].convert_dtypes()
Out[53]:
0            10,000
1           500,000
2         5,000,000
3        50,000,000
4           100,000
            ...    
10833         1,000
10834           500
10836         5,000
10837           100
10840    10,000,000
Name: Installs, Length: 7424, dtype: string
In [54]:
lc.head()
Out[54]:
App Category Rating Reviews Size Installs Type Price Content Rating Genres Last Updated Current Ver Android Ver Rating_category
0 Photo Editor & Candy Camera & Grid & ScrapBook ART_AND_DESIGN 4.1 2.201397 19000000 10,000 Free 0 Everyone Art & Design January 7, 2018 1.0.0 4.0.3 and up High
1 Coloring book moana ART_AND_DESIGN 3.9 2.985426 14000000 500,000 Free 0 Everyone Art & Design;Pretend Play January 15, 2018 2.0.0 4.0.3 and up High
2 U Launcher Lite – FREE Live Cool Themes, Hide ... ART_AND_DESIGN 4.7 4.942058 8.7000000 5,000,000 Free 0 Everyone Art & Design August 1, 2018 1.2.4 4.0.3 and up High
3 Sketch - Draw & Paint ART_AND_DESIGN 4.5 5.333737 25000000 50,000,000 Free 0 Teen Art & Design June 8, 2018 Varies with device 4.2 and up High
4 Pixel Draw - Number Art Coloring Book ART_AND_DESIGN 4.3 2.985426 2.8000000 100,000 Free 0 Everyone Art & Design;Creativity June 20, 2018 1.1 4.4 and up High

QUESTION 11:¶

Check the column 'Price' , remove the unwanted characters and convert the column into a suitable data type.

SOLUTION:¶

In [55]:
lc["Price"]
Out[55]:
0        0
1        0
2        0
3        0
4        0
        ..
10833    0
10834    0
10836    0
10837    0
10840    0
Name: Price, Length: 7424, dtype: object
In [56]:
lc["Price"].unique()
Out[56]:
array(['0', '$4.99', '$6.99', '$7.99', '$3.99', '$5.99', '$2.99', '$1.99',
       '$9.99', '$0.99', '$9.00', '$5.49', '$10.00', '$24.99', '$11.99',
       '$79.99', '$16.99', '$14.99', '$29.99', '$12.99', '$3.49',
       '$10.99', '$7.49', '$1.50', '$19.99', '$15.99', '$33.99', '$39.99',
       '$2.49', '$4.49', '$1.70', '$1.49', '$3.88', '$399.99', '$17.99',
       '$400.00', '$3.02', '$1.76', '$4.84', '$4.77', '$1.61', '$1.59',
       '$6.49', '$1.29', '$299.99', '$379.99', '$37.99', '$18.99',
       '$389.99', '$8.49', '$1.75', '$14.00', '$2.00', '$3.08', '$2.59',
       '$19.40', '$15.46', '$8.99', '$3.04', '$13.99', '$4.29', '$3.28',
       '$4.60', '$1.00', '$2.90', '$1.97', '$2.56', '$1.20'], dtype=object)
In [57]:
#changing the Unwanted Characters in the dataset 
lc["Price"] = lc["Price"].str.replace("$", "")

C:\Users\SUNILKUMARAV\AppData\Local\Temp\ipykernel_24020\1643563157.py:2: FutureWarning:

The default value of regex will change from True to False in a future version. In addition, single character regular expressions will *not* be treated as literal strings when regex=True.
In [58]:
lc["Price"].unique()
Out[58]:
array(['0', '4.99', '6.99', '7.99', '3.99', '5.99', '2.99', '1.99',
       '9.99', '0.99', '9.00', '5.49', '10.00', '24.99', '11.99', '79.99',
       '16.99', '14.99', '29.99', '12.99', '3.49', '10.99', '7.49',
       '1.50', '19.99', '15.99', '33.99', '39.99', '2.49', '4.49', '1.70',
       '1.49', '3.88', '399.99', '17.99', '400.00', '3.02', '1.76',
       '4.84', '4.77', '1.61', '1.59', '6.49', '1.29', '299.99', '379.99',
       '37.99', '18.99', '389.99', '8.49', '1.75', '14.00', '2.00',
       '3.08', '2.59', '19.40', '15.46', '8.99', '3.04', '13.99', '4.29',
       '3.28', '4.60', '1.00', '2.90', '1.97', '2.56', '1.20'],
      dtype=object)
In [60]:
# Changing  of column named Price from Object dtype to best Suitable dtype.
lc["Price"].convert_dtypes()
Out[60]:
0        0
1        0
2        0
3        0
4        0
        ..
10833    0
10834    0
10836    0
10837    0
10840    0
Name: Price, Length: 7424, dtype: string

QUESTION 12:¶

Drop the columns which you think redundant for the analysis.(suggestion: drop column 'rating', since we created a new feature from it (i.e. rating_category) and the columns 'App', 'Rating' ,'Genres','Last Updated', 'Current Ver','Android Ver' columns since which are redundant for our analysis)

In [61]:
lc.columns
Out[61]:
Index(['App', 'Category', 'Rating', 'Reviews', 'Size', 'Installs', 'Type',
       'Price', 'Content Rating', 'Genres', 'Last Updated', 'Current Ver',
       'Android Ver', 'Rating_category'],
      dtype='object')
In [62]:
lc.drop(["App","Rating","Genres","Last Updated","Current Ver","Android Ver"], axis = 1,inplace = True)
lc
Out[62]:
Category Reviews Size Installs Type Price Content Rating Rating_category
0 ART_AND_DESIGN 2.201397 19000000 10,000 Free 0 Everyone High
1 ART_AND_DESIGN 2.985426 14000000 500,000 Free 0 Everyone High
2 ART_AND_DESIGN 4.942058 8.7000000 5,000,000 Free 0 Everyone High
3 ART_AND_DESIGN 5.333737 25000000 50,000,000 Free 0 Teen High
4 ART_AND_DESIGN 2.985426 2.8000000 100,000 Free 0 Everyone High
... ... ... ... ... ... ... ... ...
10833 BOOKS_AND_REFERENCE 1.643453 619000 1,000 Free 0 Everyone High
10834 FAMILY 0.845098 2.6000000 500 Free 0 Everyone High
10836 FAMILY 1.579784 53000000 5,000 Free 0 Everyone High
10837 FAMILY 0.602060 3.6000000 100 Free 0 Everyone High
10840 LIFESTYLE 5.600218 19000000 10,000,000 Free 0 Everyone High

7424 rows × 8 columns

QUESTION 13:¶

Encode the categorical columns.

SOLUTION:¶

In [63]:
lc.dtypes
Out[63]:
Category            object
Reviews            float64
Size                object
Installs            object
Type                object
Price               object
Content Rating      object
Rating_category     object
dtype: object
In [64]:
#Obtaining the count of distinct categories from the "Category" column.
lc.Category.nunique()
Out[64]:
33
In [65]:
# Importing Label Encoder in order to perform ordinal encoding.
from sklearn.preprocessing import OrdinalEncoder
enc = OrdinalEncoder()
In [66]:
# cnverting the Category Column using Ordinal Encoding.
lc["Category"] = enc.fit_transform(lc[["Category"]])
In [67]:
lc
Out[67]:
Category Reviews Size Installs Type Price Content Rating Rating_category
0 0.0 2.201397 19000000 10,000 Free 0 Everyone High
1 0.0 2.985426 14000000 500,000 Free 0 Everyone High
2 0.0 4.942058 8.7000000 5,000,000 Free 0 Everyone High
3 0.0 5.333737 25000000 50,000,000 Free 0 Teen High
4 0.0 2.985426 2.8000000 100,000 Free 0 Everyone High
... ... ... ... ... ... ... ... ...
10833 3.0 1.643453 619000 1,000 Free 0 Everyone High
10834 11.0 0.845098 2.6000000 500 Free 0 Everyone High
10836 11.0 1.579784 53000000 5,000 Free 0 Everyone High
10837 11.0 0.602060 3.6000000 100 Free 0 Everyone High
10840 18.0 5.600218 19000000 10,000,000 Free 0 Everyone High

7424 rows × 8 columns

CONTENT RATING¶

In [68]:
# Having Unique ratings from Content Rating column.
lc["Content Rating"].unique()
Out[68]:
array(['Everyone', 'Teen', 'Everyone 10+', 'Mature 17+',
       'Adults only 18+', 'Unrated'], dtype=object)
In [70]:
#Converting the Category Column using ordinal Encoding.
lc["Content Rating"] = enc.fit_transform(lc[["Content Rating"]])
lc
Out[70]:
Category Reviews Size Installs Type Price Content Rating Rating_category
0 0.0 2.201397 19000000 10,000 Free 0 1.0 High
1 0.0 2.985426 14000000 500,000 Free 0 1.0 High
2 0.0 4.942058 8.7000000 5,000,000 Free 0 1.0 High
3 0.0 5.333737 25000000 50,000,000 Free 0 4.0 High
4 0.0 2.985426 2.8000000 100,000 Free 0 1.0 High
... ... ... ... ... ... ... ... ...
10833 3.0 1.643453 619000 1,000 Free 0 1.0 High
10834 11.0 0.845098 2.6000000 500 Free 0 1.0 High
10836 11.0 1.579784 53000000 5,000 Free 0 1.0 High
10837 11.0 0.602060 3.6000000 100 Free 0 1.0 High
10840 18.0 5.600218 19000000 10,000,000 Free 0 1.0 High

7424 rows × 8 columns

TYPE¶

In [72]:
lc["Type"].unique()
Out[72]:
array(['Free', 'Paid'], dtype=object)
In [74]:
lc["Type"] = enc.fit_transform(lc[["Type"]])
lc
Out[74]:
Category Reviews Size Installs Type Price Content Rating Rating_category
0 0.0 2.201397 19000000 10,000 0.0 0 1.0 High
1 0.0 2.985426 14000000 500,000 0.0 0 1.0 High
2 0.0 4.942058 8.7000000 5,000,000 0.0 0 1.0 High
3 0.0 5.333737 25000000 50,000,000 0.0 0 4.0 High
4 0.0 2.985426 2.8000000 100,000 0.0 0 1.0 High
... ... ... ... ... ... ... ... ...
10833 3.0 1.643453 619000 1,000 0.0 0 1.0 High
10834 11.0 0.845098 2.6000000 500 0.0 0 1.0 High
10836 11.0 1.579784 53000000 5,000 0.0 0 1.0 High
10837 11.0 0.602060 3.6000000 100 0.0 0 1.0 High
10840 18.0 5.600218 19000000 10,000,000 0.0 0 1.0 High

7424 rows × 8 columns

Rating Caterory¶

In [75]:
lc["Rating_category"].unique()
Out[75]:
array(['High', 'Low'], dtype=object)
In [77]:
lc["Rating_category"] = enc.fit_transform(lc[["Rating_category"]])
lc
Out[77]:
Category Reviews Size Installs Type Price Content Rating Rating_category
0 0.0 2.201397 19000000 10,000 0.0 0 1.0 0.0
1 0.0 2.985426 14000000 500,000 0.0 0 1.0 0.0
2 0.0 4.942058 8.7000000 5,000,000 0.0 0 1.0 0.0
3 0.0 5.333737 25000000 50,000,000 0.0 0 4.0 0.0
4 0.0 2.985426 2.8000000 100,000 0.0 0 1.0 0.0
... ... ... ... ... ... ... ... ...
10833 3.0 1.643453 619000 1,000 0.0 0 1.0 0.0
10834 11.0 0.845098 2.6000000 500 0.0 0 1.0 0.0
10836 11.0 1.579784 53000000 5,000 0.0 0 1.0 0.0
10837 11.0 0.602060 3.6000000 100 0.0 0 1.0 0.0
10840 18.0 5.600218 19000000 10,000,000 0.0 0 1.0 0.0

7424 rows × 8 columns

QUESTION 14:¶

Segregate the target and independent features (Hint: Use Rating_category as the target)

SOLUTION:¶

In [79]:
# Creating variable X with all Independent Variables.
X = lc.drop("Rating_category", axis=1)

# Creating variable Y with Dependent Variable.
Y = lc[["Rating_category"]]
In [80]:
    X
Out[80]:
Category Reviews Size Installs Type Price Content Rating
0 0.0 2.201397 19000000 10,000 0.0 0 1.0
1 0.0 2.985426 14000000 500,000 0.0 0 1.0
2 0.0 4.942058 8.7000000 5,000,000 0.0 0 1.0
3 0.0 5.333737 25000000 50,000,000 0.0 0 4.0
4 0.0 2.985426 2.8000000 100,000 0.0 0 1.0
... ... ... ... ... ... ... ...
10833 3.0 1.643453 619000 1,000 0.0 0 1.0
10834 11.0 0.845098 2.6000000 500 0.0 0 1.0
10836 11.0 1.579784 53000000 5,000 0.0 0 1.0
10837 11.0 0.602060 3.6000000 100 0.0 0 1.0
10840 18.0 5.600218 19000000 10,000,000 0.0 0 1.0

7424 rows × 7 columns

In [81]:
Y
Out[81]:
Rating_category
0 0.0
1 0.0
2 0.0
3 0.0
4 0.0
... ...
10833 0.0
10834 0.0
10836 0.0
10837 0.0
10840 0.0

7424 rows × 1 columns

QUESTION 15:¶

Split the dataset into train and test.

SOLUTION:¶

In [82]:
 #importing library for split  the dataset.
from sklearn.model_selection import train_test_split
In [83]:
#Spliting the data set into Train and Test.
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.30 , random_state=1)
In [84]:
X_train
Out[84]:
Category Reviews Size Installs Type Price Content Rating
8888 0.0 0.000000 2.9000000 100 0.0 0 1.0
1298 15.0 5.264019 57000000 5,000,000 0.0 0 1.0
6599 20.0 2.489958 5.9000000 10,000 0.0 0 1.0
7328 30.0 2.328380 6.1000000 5,000 1.0 3.99 1.0
5681 18.0 1.755875 5.8000000 10,000 0.0 0 1.0
... ... ... ... ... ... ... ...
1522 17.0 2.731589 99000000 50,000 0.0 0 1.0
7755 11.0 1.462398 37000000 1,000 0.0 0 2.0
5978 11.0 0.000000 2.0000000 50 1.0 1.49 1.0
347 6.0 6.679963 11000000 500,000,000 0.0 0 1.0
7698 14.0 0.698970 12000000 100 0.0 0 1.0

5196 rows × 7 columns

In [85]:
X_test
Out[85]:
Category Reviews Size Installs Type Price Content Rating
5708 29.0 2.845718 7.3000000 100,000 0.0 0 1.0
7215 11.0 2.559907 14000000 1,000 1.0 2.99 2.0
1350 15.0 4.852901 49000000 1,000,000 0.0 0 1.0
5727 29.0 4.895583 6.3000000 1,000,000 0.0 0 1.0
5326 11.0 3.997910 3.7000000 100,000 0.0 0 4.0
... ... ... ... ... ... ... ...
6351 29.0 3.066699 350000 100,000 0.0 0 1.0
1634 18.0 3.493319 13000000 100,000 0.0 0 1.0
9865 3.0 1.518514 37000000 1,000 0.0 0 1.0
10396 11.0 5.345748 94000000 5,000,000 0.0 0 4.0
6550 11.0 1.812913 24000000 10,000 0.0 0 1.0

2228 rows × 7 columns

In [86]:
Y_train
Out[86]:
Rating_category
8888 0.0
1298 0.0
6599 0.0
7328 0.0
5681 0.0
... ...
1522 0.0
7755 0.0
5978 1.0
347 0.0
7698 0.0

5196 rows × 1 columns

In [87]:
Y_test
Out[87]:
Rating_category
5708 0.0
7215 0.0
1350 0.0
5727 0.0
5326 0.0
... ...
6351 0.0
1634 0.0
9865 0.0
10396 0.0
6550 0.0

2228 rows × 1 columns

QUESTION 16:¶

Standardize the data, so that the values are within a particular range

SOLUTION:¶

In [88]:
#Importing the libraries to Standardize the data.
from sklearn.preprocessing import StandardScaler
In [89]:
lc.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 7424 entries, 0 to 10840
Data columns (total 8 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   Category         7424 non-null   float64
 1   Reviews          7424 non-null   float64
 2   Size             7424 non-null   object 
 3   Installs         7424 non-null   object 
 4   Type             7424 non-null   float64
 5   Price            7424 non-null   object 
 6   Content Rating   7424 non-null   float64
 7   Rating_category  7424 non-null   float64
dtypes: float64(5), object(3)
memory usage: 522.0+ KB

¶

As the columns labeled Size, Installs, and Price are currently in the object data type, converting them to integer data type is necessary to carry out standardization

In [90]:
lc['Installs'] = lc['Installs'].apply(lambda x : x.replace(',','').replace('+','')).astype(int)
lc['Size'] = lc['Size'].apply(lambda x : x.replace(',','')).astype(float)
lc['Price'] = lc['Price'].apply(lambda x : x.replace(',','')).astype(float)
In [91]:
scaler = StandardScaler()
In [92]:
# Performing Standardization of data
lc = scaler.fit_transform(lc)
In [93]:
lc
Out[93]:
array([[-2.03766618, -0.67995305, -0.07422317, ..., -0.06305842,
        -0.46322046, -0.35358018],
       [-2.03766618, -0.19653147, -0.27499084, ..., -0.06305842,
        -0.46322046, -0.35358018],
       [-2.03766618,  1.0099001 , -0.83713998, ..., -0.06305842,
        -0.46322046, -0.35358018],
       ...,
       [-0.68621673, -1.0632313 ,  1.29099702, ..., -0.06305842,
        -0.46322046, -0.35358018],
       [-0.68621673, -1.66608206, -0.83714018, ..., -0.06305842,
        -0.46322046, -0.35358018],
       [ 0.17379656,  1.41571255, -0.07422317, ..., -0.06305842,
        -0.46322046, -0.35358018]])
In [94]:
lc = pd.DataFrame(lc)
lc
Out[94]:
0 1 2 3 4 5 6 7
0 -2.037666 -0.679953 -0.074223 -0.168764 -0.282029 -0.063058 -0.463220 -0.35358
1 -2.037666 -0.196531 -0.274991 -0.158181 -0.282029 -0.063058 -0.463220 -0.35358
2 -2.037666 1.009900 -0.837140 -0.060990 -0.282029 -0.063058 -0.463220 -0.35358
3 -2.037666 1.251404 0.166698 0.910911 -0.282029 -0.063058 2.517665 -0.35358
4 -2.037666 -0.196531 -0.837140 -0.166820 -0.282029 -0.063058 -0.463220 -0.35358
... ... ... ... ... ... ... ... ...
7419 -1.669089 -1.023974 -0.812285 -0.168958 -0.282029 -0.063058 -0.463220 -0.35358
7420 -0.686217 -1.516228 -0.837140 -0.168969 -0.282029 -0.063058 -0.463220 -0.35358
7421 -0.686217 -1.063231 1.290997 -0.168872 -0.282029 -0.063058 -0.463220 -0.35358
7422 -0.686217 -1.666082 -0.837140 -0.168977 -0.282029 -0.063058 -0.463220 -0.35358
7423 0.173797 1.415713 -0.074223 0.046999 -0.282029 -0.063058 -0.463220 -0.35358

7424 rows × 8 columns

In [96]:
lc.columns=['Category','Reviews','Size','Installs','Type','Price','Content Rating','Rating_Category']
lc
Out[96]:
Category Reviews Size Installs Type Price Content Rating Rating_Category
0 -2.037666 -0.679953 -0.074223 -0.168764 -0.282029 -0.063058 -0.463220 -0.35358
1 -2.037666 -0.196531 -0.274991 -0.158181 -0.282029 -0.063058 -0.463220 -0.35358
2 -2.037666 1.009900 -0.837140 -0.060990 -0.282029 -0.063058 -0.463220 -0.35358
3 -2.037666 1.251404 0.166698 0.910911 -0.282029 -0.063058 2.517665 -0.35358
4 -2.037666 -0.196531 -0.837140 -0.166820 -0.282029 -0.063058 -0.463220 -0.35358
... ... ... ... ... ... ... ... ...
7419 -1.669089 -1.023974 -0.812285 -0.168958 -0.282029 -0.063058 -0.463220 -0.35358
7420 -0.686217 -1.516228 -0.837140 -0.168969 -0.282029 -0.063058 -0.463220 -0.35358
7421 -0.686217 -1.063231 1.290997 -0.168872 -0.282029 -0.063058 -0.463220 -0.35358
7422 -0.686217 -1.666082 -0.837140 -0.168977 -0.282029 -0.063058 -0.463220 -0.35358
7423 0.173797 1.415713 -0.074223 0.046999 -0.282029 -0.063058 -0.463220 -0.35358

7424 rows × 8 columns

PART 2¶

DATA VISUALIZATION ON HONEY PRODUCTION DATASET USING SEABORN AND MATPLOTLIB¶

QUESTION 1:¶

Import required libraries and read the dataset.

SOLUTION:¶

In [97]:
import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np

QUESTION 2:¶

Check the first few samples, shape, info of the data and try to familiarize yourself with different features.

SOLUTION:¶

In [99]:
lc1=pd.read_csv("honeyproduction.csv")
lc1
Out[99]:
state numcol yieldpercol totalprod stocks priceperlb prodvalue year
0 AL 16000.0 71 1136000.0 159000.0 0.72 818000.0 1998
1 AZ 55000.0 60 3300000.0 1485000.0 0.64 2112000.0 1998
2 AR 53000.0 65 3445000.0 1688000.0 0.59 2033000.0 1998
3 CA 450000.0 83 37350000.0 12326000.0 0.62 23157000.0 1998
4 CO 27000.0 72 1944000.0 1594000.0 0.70 1361000.0 1998
... ... ... ... ... ... ... ... ...
621 VA 4000.0 41 164000.0 23000.0 3.77 618000.0 2012
622 WA 62000.0 41 2542000.0 1017000.0 2.38 6050000.0 2012
623 WV 6000.0 48 288000.0 95000.0 2.91 838000.0 2012
624 WI 60000.0 69 4140000.0 1863000.0 2.05 8487000.0 2012
625 WY 50000.0 51 2550000.0 459000.0 1.87 4769000.0 2012

626 rows × 8 columns

In [100]:
lc1.head(5)
Out[100]:
state numcol yieldpercol totalprod stocks priceperlb prodvalue year
0 AL 16000.0 71 1136000.0 159000.0 0.72 818000.0 1998
1 AZ 55000.0 60 3300000.0 1485000.0 0.64 2112000.0 1998
2 AR 53000.0 65 3445000.0 1688000.0 0.59 2033000.0 1998
3 CA 450000.0 83 37350000.0 12326000.0 0.62 23157000.0 1998
4 CO 27000.0 72 1944000.0 1594000.0 0.70 1361000.0 1998
In [101]:
lc1.shape
Out[101]:
(626, 8)
In [102]:
lc1.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 626 entries, 0 to 625
Data columns (total 8 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   state        626 non-null    object 
 1   numcol       626 non-null    float64
 2   yieldpercol  626 non-null    int64  
 3   totalprod    626 non-null    float64
 4   stocks       626 non-null    float64
 5   priceperlb   626 non-null    float64
 6   prodvalue    626 non-null    float64
 7   year         626 non-null    int64  
dtypes: float64(5), int64(2), object(1)
memory usage: 39.3+ KB

# The dataset comprises a single column labeled 'State', which is of object type, while all other columns are numerical. The dataset has a shape of (626, 8), consisting of 626 rows and 8 columns.

QUESTION 3:¶

Display the percentage distribution of the data in each year using the pie chart.

SOLUTION:¶

In [105]:
dist = lc1.groupby('year').size()/len(lc1)*100
dist
Out[105]:
year
1998    6.869010
1999    6.869010
2000    6.869010
2001    7.028754
2002    7.028754
2003    7.028754
2004    6.549521
2005    6.549521
2006    6.549521
2007    6.549521
2008    6.549521
2009    6.389776
2010    6.389776
2011    6.389776
2012    6.389776
dtype: float64
In [106]:
plt.pie(dist, labels=dist.index,autopct='%1.1f%%')
plt.title("Distribution")
plt.show()

QUESTION 4:¶

Plot and Understand the distribution of the variable "price per lb" using displot, and write your findings.

SOLUTION:¶

In [107]:
sns.displot(data=lc1['priceperlb'])
Out[107]:
<seaborn.axisgrid.FacetGrid at 0x243e182c310>

COMMENT¶

The longest-duration average price is observed within the range of 1.1 to 1.5, indicating a gradual upward trend.

QUESTION 5:¶

Plot and understand the relationship between the variables 'numcol' and 'prodval' through scatterplot, and write your findings.

SOLUTION:¶

In [108]:
plt.scatter(lc1['numcol'], lc1['prodvalue'])
plt.xlabel('Number of Colonies')
plt.ylabel('Production Value')
plt.title('numcol vs prodval')
plt.show()

COMMENT

In my observation, there appears to be a positive correlation between the increase in the number of colonies and the rise in production value.

QUESTION 6:¶

Plot and understand the relationship between categorical variable 'year' and a numerical variable 'prodvalue' through boxplot, and write your findings.

SOLUTION:¶

In [109]:
sns.boxplot(x=lc1['year'],y=lc1['prodvalue'],linewidth=None)
plt.figure(figsize=(50,50))
plt.show()

<Figure size 5000x5000 with 0 Axes>

COMMENT¶

PRODUCTION VALUE HIGH IN 2010

QUESTION 7:¶

Visualize and understand the relationship between the multiple pairs of variables throughout different years using pairplot and add your inferences. (use columns 'numcol', 'yield percol', 'total prod', 'prodvalue','year')

SOLUTION:¶

In [110]:
sns.pairplot(lc1, vars=['numcol', 'yieldpercol', 'totalprod', 'prodvalue', 'year'], hue='year')
Out[110]:
<seaborn.axisgrid.PairGrid at 0x243e7a16f10>

COMMENT¶

A linear correlation is identified between the variable pairs "prodval" and "numcol," as well as "prodval" and "totalprod."

QUESTION 8:¶

Display the correlation values using a plot and add your inferences. (use columns 'numcol', 'yield percol', 'total prod', 'stocks', 'price per lb', 'prodvalue')

In [111]:
corr_matrix=lc1[['numcol', 'yieldpercol', 'totalprod', 'stocks', 'priceperlb', 'prodvalue']].corr()
In [112]:
corr_matrix
Out[112]:
numcol yieldpercol totalprod stocks priceperlb prodvalue
numcol 1.000000 0.243515 0.953594 0.825929 -0.232701 0.912796
yieldpercol 0.243515 1.000000 0.396252 0.367812 -0.358646 0.278977
totalprod 0.953594 0.396252 1.000000 0.878830 -0.264499 0.907236
stocks 0.825929 0.367812 0.878830 1.000000 -0.305867 0.728560
priceperlb -0.232701 -0.358646 -0.264499 -0.305867 1.000000 -0.089567
prodvalue 0.912796 0.278977 0.907236 0.728560 -0.089567 1.000000
In [113]:
#plotting the heatmap
sns.heatmap(corr_matrix, annot=True)
plt.title("Correlation Matrix")
plt.show()

COMMENT¶

The variables 'totalprod' and 'numcol' exhibit the strongest positive correlation, while 'Priceperlb' and 'stocks' display the most pronounced negative relationship between them.

In [ ]: